Papers with PPTX-Match Evaluation System
PPTC Benchmark: Evaluating Large Language Models for PowerPoint Task Completion (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent evaluations of Large Language Models (LLMs) focus on their zero-shot/few-shot capabilities for basic natural language tasks and their ability to translate instructions into tool APIs. |
| Approach: | They propose a PowerPoint Task Completion benchmark to assess LLMs’ ability to create and edit PPT files based on user instructions. |
| Outcome: | The proposed system outperforms open-source and closed LLMs with 75.1% accuracy in single-turn dialogue testing but only achieves 6% session accuracy. |